Papers with model compression approach
Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using dynamic slicing, large language models can be used to reduce computational burden and improve performance. |
| Approach: | They propose a dynamic layer-specific pruning approach that leverages the newly proposed Layer Redundancy score to prune parts of individual layers based on redundancy. |
| Outcome: | The proposed method maintains and enhances model performance over the SliceGPT baseline. |
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to compress neural networks by progressive module replacement is proposed . a number of techniques have been proposed to compress pretraining and fine-tuning models . |
| Approach: | They propose a model compression approach that divides BERT into modules and builds their compact substitutes. |
| Outcome: | The proposed approach outperforms existing knowledge distillation approaches on GLUE benchmark . it is based on a model that divides the original BERT into several modules and builds their substitutes . |